Di erent Boosting Algorithms and Underlying Optimization Problems

Boosting is an ensemble based method which attempts to boost the accuracy of any given learning algorithm by applying it several times on slightly modi ed training data and then combining the results in a suitable manner. The boosting algorithms that we covered in class were AdaBoost, LPBoost, TotalBoost, SoftBoost, and Entropy Regularized LPBoost. The basic idea behind these boosting algorithms is that at each iteration, a weak learner learns the training data with respect to a distribution. The weak learner is then added to the nal strong learner. This is typically done by weighting the weak learner in some manner, which is typically related to the weak learner's accuracy. After the weak learner is added to the nal strong learner, the data is reweighted: examples that are misclassi ed gain weight and examples that are classi ed correctly lose weight. Thus, future weak learners will focus more on the examples that previous weak learners misclassi ed.